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Comparison of CEAS and WllUams-Type Barley Yield Models fbr North 
Dakota and Minnesota. By Sharon LeDuc, NCAA/ED IS/CEAS, Models Brandi, 
Colunbla, Missouri. 


ABSTOACT 


Ihe nodels ha/e each been evaluated In separate documents (Barnett, 
1981 ). The models are compared based cai specified select ic»i criteria v*ilch 
Includes a ten ^ar bootstrap test (1970-1979). Based on this the models 
were quite comparable. However, tiie CEAS model was slig^itly better 
overall. The Wlllians-type model seemed better fbr the 1974 estimates. 
Because that year spring vftieat yldLd was particularly low ttie WilUans-type 
model should not be excluded from further ccxislde ration. 
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Comparison of CEAS azvl WUUanis-Type Barley Nsdsls fbi 
North Dakota and Minnesota 

by Sharon LeDuc 

Review of Models 

Ihe CEAS (Motha, 1980) and the MUUams-'I^ype (Williams 1975) tjarley 
models are nultlple regression nodels vlilch use nonthly mean tonperature 
and monthly total precipitation data as the basis for develqplng predictor 
variables. The tenqperature and precipitation are the average for all st»> 
tlons within a crop reporting district. Information on the models is 
available In two papers : "Evaluation (EAS Barley Model in North Dakota 
and Minnesota" and "Evaluation of Williams l^pe Barley Model In North 
Dakota and Minnesota" (Barnett, 1981). The WilUams-'l^pe model was designed 
to parallel the models by Williams (1975) but still be comparable to the 
CEAS models. lt*end is handled Identically in the two models as a linear 
and quadratic fixictlon of year. The distlr^ulshlng differences are the 
inclusion of soil texture and topographical variables in the Williams ^pe 
model and the subsequent pooling of cross sectional data (the separate crop 
reporting districts) in selecting predictor variables and estimating the 
coefficients. 

Oomparls<xi Methodology 
Elrfit Model Ooaracteristics to be Compared 
The document, Crop Yield Model Test and Evaluation Criteria . (Wilson et 
al., 1980), states: 

The model characteristics to be emphasized in the evaluation process 
are: yld.d Indication reliability, ob.lectlvity, c(»iDistency vdth 
scientific knowledge, adequacy, timeliness, minimum cost, slnplicity, 
and accurate current measures of modeled yield reliability. 
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Figure 1. Production of barley by CRD (1970-79 average) aa a percent of the regional total. Darker ahadea 
Indicate CKOs with higher production. 
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The nodels xUl )3e cgopared using these characteristics; each is 
discussed Ijidividually Mithout regard to the other characteristics. The 
present discussion natos no presunpticMi as to the relative Importance of 
the characteristics. 

Qugititative Model Oompariscyis 
Are on the Same Date 

Dirsct quantitative comparisons between nodels will be made fbr two of 
the previously mentioned criteria: (1) yield Indication reliability 
(2) accurate current neasuree of nodeled yield reliability. The quajititles 
Involved are derived from the observed yields and the nodel predicted 
ylel'Js and standard errors of predlctlm obtained fron Independent 
bootstrap testa fbr each of ten years (1970-1979) • Ihe sause base period 
is used for all models In computing model related values fbr a particular 
year. The a*erage productlm and yield ever the ten year test period are 
listed In Table 1 for each geogrephlc area. Also shawn is the percent 
production each crop reportlr^ district (CHD) contributes to its state and 
the three state region and the percent proeixtion each state contributes 
to the region. The percentage of regional production fbr each CH) is shovn 
graphically In Figure 1. Darker shades indicate higher productivity. 
Separate models are derived fbr each CRD, state, and the region. Model 
related values (predictions and standard errors of prediction) at the 
state level are also obtained by using a wel^ta* average of that state's 
CRD model values. Model related values fbr the region are also obtained 
using a wei^ted mrerage of the values frem the CRD models and from the 
state models. Ihe wel^tlng factor used Is harvested acreage. Results 
obtained by aggrefi^iting from the CRD nodels are Identified In tables as 
"CRD AGCS^." Results obtained by aggregating from the state models are 
Identified as "STATES A3C51." 
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Li addltlm tx> the CEAS and WUliams-type nodal, the "straMUBn” nodal 
la Included In Tablet^ 2-10, 12-14 and In Appa.ndix 1. Ihla nodal type la 
deacribed by Sebau^ (1961) and will not be dlacuaaed here. 

K ^la Are Hanked According to Performance 


Nodela are ranked fbr each of the ibllowing Indicators of yield reliability 
(order does not laf}ly rdLative Importance): 

(1) the bias, 

(2) the root mean square error (RMSE), 

(3) the standard deviation (SD), 

(4) the percent of years the absolute value of the relative dlfferenoi 
exceeds te.i percent, 

(3) the largest absolute value tte relative difference, 

(6) the next largest absolute value of tl'ie relative difference, 

(7) the percent of jears In »dilch the direct icm of change from the 
previous year In the ?'s agrees with the Y's. 

(8) the percent of years in vhlch the dlrectlit»i.of chang^e from the 
a^3ragc of the previous three years In the Y's agrees with the 
Y's, and 

(9) the Pearson correiatlCTi coefficient between ttie actual and pre- 
dicted yj.rids during the Independent test jiears. 

Models are au.se ranked according to the value of the Spearman correlation 
coefficient vhlch Indicates the utility of the nodel's current measure of 
modeled yl.dd reliability. R>r nost of the indicators (1-6), the nodel 
with the oiBllest nuneric value exhibits the best performance In terms of 
yield reliability and is given a rank of 1. Fbr the remaining quantities, 
the model with the largest value exhibits the most desirable performance. 

If models ar*e tied for the same level of performance, they are all assl^ied 
the lowest rank fbr »*ilch they aire tied. R>r example, if two models aire 
tied for best performance, Uoey are both asslgied a rank of 1, the lower of 
ranks 1 and 2. 

It should be raroembered that the models are tanked only In relation to each 
other and rx>t to an absolute standard. Therefore, saying that a particular 
model performs better, is superior to, or more desirable than another model 
does not necessarily In^ly that thie model is the best of all possible 
models. It Is the best of only those with which it is currently being 
con^ared. 
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tfodelB are Oonpared Ualng Statxatioal 

A 

Teata Baaed on d ■ Y - Y 


It la dsslrable to run a atatlatlcal teat o(arpari!ig the r^labllity of ccro- 
peting nndels. A fbnnal atatlatlcal teat cc^ldera the variability of 
model performance ever time and allows the user to specify an upper limit 
on the ppotablllty of Incorrectly declaring one nodel better than another. 
*nila probability la Inovn aa a, the level of sl0ilflsance, or the Type I 
error. 

However, because of t^ie nanner In ihlch models are chosofi fbr testing and 
how they are evaluated. It la challenging to construct a meaningful sta- 
tistical test. Only yield models vtilch have been presented In the lltera?- 
ture or developed by hiowri experts are ccxisldered. Therefore, a priori, 
great differences between the rellablll^ of the modsls are not expected. 

A pwerful statistical proced.\re la needed which la able to detect fvnall, 
although Inqsortant, differences in reliability. Also, the test should be 
able to finctlon well with relatively atnall samples of data for each model, 
say ten yeairs. 

The test should also perform well vhen only two models are being cempared. 
Often only two models of a particular type, Ibr exaii^jle, two monthly 
weather ebta models or two dally weather data nodels, are ccmpetltlve and 
available Ibr testing. Vhen models of dlffermt types are to be coropai*ed. 
It Is unlikely that all possible model comparisons will be made. It Is 
more likely that the best models of eah type will be compared. 

It would appear that an P test could be useful in comparing the mean square 
errut's of two models. However, if the mean square errors are baaed on ten 
years of test data and « ■ .05, then one irodel’s mean square error nust be 
four times larger than anothers before the nodels can be declared 
different. This Is an unreasonable requirement since nodels vhlch are In 
the evaluation process will almost always be more competitive than this. 

A test may be constructed by considering that one nodgl la ccxialdered more 
reliable than another model If Its predicted yields, Y's, are closer to the 
actual yields, Y’s. No difference In the rrilabillty of two nodals fbr a 
particular yrar means that the absolute value of the difference between 
their predicted yields and the actual yield Is the same. The absolute 
value of the difference la used because It does not natter vliether cne 
model over-es time tee and the other underestimates or whether they both over 
or under-estlmate. The reliability of a model fbr that year Is related to 
the anount of the discrepancy, .not its direction. We may define 
ld,| - |Yi - Y|, dal - |Yj - Yl . md D - I dj] - |dj| . 
mocbls are equall;y reliable In a year fbr which D ^uals zero. If D Is not 
equal to zero, one model Is more reliable than the other fbr that year. Th 
formal terms, we want to test the rul.' hypothesis that there Is no dif- 
ference In the reliability of the models ever all years. Tb do so the 
values of D from the ten test years may be used to cempute a test statistic 
and a decision m vhether or not to reject the mill hypothesis. Since 

the rraults fbr the models are paired each year, paired-sample statistical 
tests are used. 
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Tvo types of taired-san^ae statistical tests are used: a parametric test 
using the s^ lent t-test statistic and a nonparametrlc test using the 
Wllcoron sl®ied rank test statistic. Che reason fbr applyli^ both tests Is 
that they require different assun?)tlons . The parametric t-teet assunss the 
D values are nornally distributed while the noparametrlc test does not, 

•me d values may be considered to be approximately nor mal ly distributed. 

Ihe I d I values would then be fblded normals rather than notmally 
distributed. Althou^ both models are Iblded at ^ d | = 0, their means may 
be different and the distribution of D has a possibility of not belr« nor- 
mally distributed. The t-test Is robust with respect to the notnallty 
assumption; however, this possible violation of the assumption Is one 
reason fbr also running the non-parametrlc test. 

The other reason for runnli^ both tests cai<»tns the ccmdltlons under vhlch 
the mil 1 -pothesls Is rejected by each test. Using the parametric test, 
the basis for ”ejectlng the mil hypothesis Is the average size of the D 
values as compared to their variability. The b-test statistic Is the 
average of the sample D’s divided by the sample standard error cf the D's, 
The hypotl^ls will be rejected and the model with the analler \ d j values 
declaimed more reliable If t Is large (either positive or ne^tlve). 
ifowever. It Is possible that one model could hare a sraallern ^ I value for 
each of the test jears. In other words, be very consistent In outperforming 
the other model, and still the mil hyi^thesls nay not be rejected by the 
parametric test unless the average value of D Is large enou^. The tora- 
netrlc test Inplicltly requires that one model have more years with 
I d I values smaller by a sufficient amount before that model may be 
declaimed more reliable. 

Using the norparametrlc test, the null hypothesis will always be rejected 
if one nodel has smaller | d j values for each of the test years, regardless 
of the magnitude of the D values. Therefore, if the models are very ccr- 
petltlve In terras of the | d \ values each year, but one model consistently, 
although sll^tly, outperforms the other model, the ncrpararretrlc test will 
still declare the consistent model to be more reliable. 

Tlie hypothesis of equal model performance will oily be rejected by the non- 
parametrlc test if one .aodel has rore years with smaller | d | values than 
the other model. The movtel with more smaller | d | values is considered the 
more reliable model In terms of consistency of performance. However, to 
reject the mil hypothesis and declare one model clearly better than 
anotlier, consistency of performance is a necessary, but not a sufficient, 
req’iireraent . Consider the situation In which one model is more ccxos/^tent 
than the other but the largest D values occur viien the less coislstait 
model performs better. Ih the few years tiie less ccxisistent model performs 
better. It performs mch better, A dilemma exists since caie model is more 
consistent than tbs ccher but the biggest dLffei*ences betweai the models 
occur vhai the consistent model performs worse. The mil hypothesis will 
not be rejected and the consistent model will not be declared better If 
this situation occurs. The mil hypothesis will be rejected only If one 
model is more consistent and the biggest differences betweoi the models 
occiu* viien the consistent model performs better. 
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MDDEL OOMPARISON 

Indicators of Yield Reliability Based on 
d » Y - Y Show Wmiains-type Preferred for CRD Ifedels. 
but CEAS Models Better for Aggsregsated Estimates 

msdel values and ccmparatlve ranks fbr the bias, the root mean 
square error (RMSE) , and the standard deviation (SD) are given In Tables 
2, 3> snd 4. Most (all but 4) Individual CRD estimates frcm the Willi ana 
type model are less biased than the estimates ft*om the CEAS models. Hie 
CEAS state nodel estimates and regional estimates aggregated frcm the CH) 
estimates are less biased than those flrem the Williams type model (Table 2). 

Hie results shorn In Table 3 Indicate smaller RMSE for eight CR)'s vtlth 
the Wnilams-type model. Figure 2; for the CRD’s aggregated to state or 
regional values the CEAS model had smaller RMSE. The WUllams type model 
generally has smaller standard deviations than the CEAS models (Table 4). 
The CEAS aggregated CRDs had smaller standard deviations Ibr Minnesota and 
the region. Che reason is that the CEAS CFD models are better In ND 30 ani 
MN 10, both hl^ production areas. 

Indicators of Yield Reliability Based 
rd = (d/Y* 100 Show Williams TVpe Slightly Better 
for CRD estimates 


The model values and comparative ranks for the Indicators of yield 
reliability based on the relative difference, rd, are given In Tables 5,6, 
and 7. These Indicators are valuable for demonstrating the worst perfor- 
mance of a model. Therefore, the better performing model will ha^e the 
smaller value for percentage years vhen the absolute value of the rdntlye 
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table 2 

MODEL COMPARISON BASED ON THE 
BIAS (QUINTALS/HECTARE) 
DERIVED FROM INDEPENDENT TEST YEARS 


TREND and monthly WEATHER DATA MODELS 
barley 

NORTH DAKOTA AND MINNESOTA 


STATE 

CRD 1 

strawman 

BIAS RANK 

MODEL 

1 WILLIAMS 

1 BIAS RANK 

1 CEAS 

I BIAS rank 

1 

N. DAKOTA 10 

2 , 

02 

(3) 

1 

-1.06 


1 1.61 

(2) 


20 1 

1. 

68 

(3) 

- 0.45 

(1) 

1.30 

(25 


30 1 

0. 

47 

(1) 

-1.15 

(3) 

1 0.95 

(2) 


40 i 

I. 

02 

(1) 

-1.40 

(25 

1 -1.81 

(35 


59 » 

1. 

96 

(3) 

-1.28 

(1) 

1 -1.50 

(2) 


60 t 

0. 

74 

(1) 

-3.31 

(3) 

1 0.75 

(2) 


75 « 

2. 


(3) 

1 -0.56 

(1) 

1 -0.63 

(25 


80 1 

1. 

76 

(3) 

1 1.07 

(2) 

1 -0.41 

(1) 


90 i 

2. 

31 

(2) 

1 -2.95 

1 

(3) 

1 -1.25 

1 

(1) 

STATE 1 

1 

MODEL 1 

1. 

16 


• -2.31 

(35 

! 

(25 

CRDS 

ACGR. 1 
1 

I* 

12 

(2) 

1 -1.75 

(3) 

1 0.40 

1 

(15 

MINNESOTAIO 1 

0. 


U) 

1 -3.06 

(3) 

-2.24 



40 1 

2. 

86 

(35 

1 -0.87 

1 

(1) 

-2.09 

(25 

STATE 1 

1 

MODEL 1 

1. 

00 

(1) 

1 -1.90 

(3) 

-1.85 

(25 

CRDS 

AGGR, 1 

1 . 

03 

(1) 

1 -2.38 

1 

(35 

-2.09 

(25 

REGION 

1 

1 




1 

\ 




CRDS 

AGGR . 1 

1. 

07 

(2) 

1 -1.95 

(35 

-0.28 

(15 

STATES 

AGGR, 1 

1. 

12 

(1) 

1 -2.20 

(3) 

-1.95 

(25 
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table 3 

MODEL COMPARISON BASED ON THE 
ROOT MEAN SQUARE ERROR (QUINTALS/HECTARE) 
DERIVED FROM INDEPENDENT TEST YEARS 

TREND AND MONTHLY WEATHER DATA MODELS 

barley 

NORTH DAKOTA AND MINNESOTA 


STRAWMAN 


model 

WILLIAMS 


CEAS 


STATE CRD | 

RMSE 

RANK 

RMSE 

RANK 

RMSE 

rank 

N. DAKOTA 10 | 

4.19 

(3) 

2.95 

(1) 

2.99 

(2) 

20 1 

3.67 

(3) 

2.65 

(1) 

3.39 

(2) 

30 1 

I'M 

?3) 

3.02 

(2) 

2.40 

(1) 

40 1 

(2) 

3.05 

(1) 

5.01 

(3) 

50 1 

4.34 

(3) 

2.73 

U) 

4.11 

(2) 

60 i 

3.50 

(2) 

4.41 

(3) 

3,26 

(1) 

70 1 

3.62 

(2) 

2.32 

(1) 

3.90 

(3) 

80 i 

4.04 

(3) 

2.67 

(1) 

3.64 


90 t 

4.33 

(3) 

3.46 

(1) 

3.60 

(2) 

STATE MODEL 1 
CRDS AGGR.I 

3.12 

(1) 

3.30 

<21 

3.35 

(3) 

3.10 

(3) 

. 

• 

2.80 

(2) 

2,44 

ID 

MINNESOTAIO | 

2.84 

(2) 


(3) 

2.55 

U’ 

40 1 
1 

5.60 

(3) 

(1) 

4.89 

12) 

STATE MODEL 1 
CRDS AGGR.I 
1 

2.97 

2.96 

(3) 

(2) 

I'M 

<2) 

(3) 

2.89 

2.65 

(1) 

ID 

REGION 1 





2.09 

1 1 ) 

CRDS AGGR.I 

2.94 

(3) 

2.85 

(2) 

STATES AGGR.I 

2.92 

<1) 

3.07 

(2) 

3.12 

(3) 
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Figure 2. Number iodlcates the model with smallest root mean square error for barley yields based on test 
years 1970-1979. Darker shades Indicate CRDs with higher production. 
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table 4 

MODEL COMPARISON! BASED ON THE 
STANDARD DEVIATION ( QUINTAlS/HECTARE ) 
DERIVED FROM INDEPENDENT TEST YEARS 

TREND AND MONTHLY WEATHER DATA MODELS 

barley 

NORTH DAKOTA AND MINNESOTA 


STATE CRD 

strawman 

SD RANK 

MODEL 

williams 

SO RANK 

CEAS 

SO rank 

N. DAKOTA 10 

3.67 

(3) 

2.76 

(2) 

2.52 

(1) 

20 

3.26 

(3) 

2,62 

(1) 

3.14 

(2) 

30 

3.12 

(3) 

2.79 

(2) 

2.21 

(1) 

40 

4.24 

(2) 

2.71 

(1) 

4,67 

(3) 

SO 

3.87 

(3) 


(1) 

3.82 

(2) 

60 

3.42 

(3) 

2.92 

(1) 

3.18 

(2) 

70 

2.90 

(2) 

2.24 

(1) 

3.84 

(3) 

80 

3.64 

(3) 

2.44 

(1) 

3.61 

(2) 

90 

3.66 

(3) 

1.81 

(1) 

3.37 

(2) 

STATE MODEL 
CRDS AGGR, 


(3) 

2.36 

(1) 

2.72 

!i! 

2.89 

13) 

2.18 

(1) 

2.40 

minnesotaio 

2.84 

(2) 

2.89 

(3) 

1.21 


40 

4.81 

(3) 

2.94 

(1) 

4,42 

(2) 

STATE MODEL 

2.80 

(3) 

2.28 

(2) 

2,22 

(1) 

CRDS AGGR. 

2.77 

13) 

2,34 

(2) 

1.62 

(1) 

REGION 

2.74 






CRDS AGGR, 

(3) 

2.09 

(2) 

2.07 

(1) 

STATES AGGR. 

2.70 

(3) 

2.14 

(1) 

2.44 

(2) 


ORIGINAL P:i;:L 
OF POOR guAimr 

difference exceeds ten percent. The largest and the next largest absolute 
value of the r^atlve difference should also be as small as possible with 
the preferred nodel. In 6 of tte 11 OTD's the Williams type nDctel Is 
within 10/C of tlie r^atlve difference more often than Is the CEAS model 
(Figure 3)« I*hr this attribute the CRD estimates ag®?egated to state and 
regional estimates are better with the CEAS model. Table 5. 

With regard to the largest absolute rriative difference (Figure 4), the 
WUllams-type nndel estimates are better In CRD 40 In Wnnesota and In all 
of North Datota (except (H) 90) than the estimates from the CEAS models. 

The aggregated CEAS CRD models are better than the aggregated WllHams-type 
models only fbr Minnesota (Table 6) . This Is also true with the next 
largest absolute relative diffeirences (Figure 5 and Table 7). Chly 3 CH)s 
show CEAS being better. However, 2 are the higher production areas so the 
regional aggregation of CEAS CRD models is better. With the CEAS modal the 
largest relative differences are all positive for North Dalcota. The 
WUliams-type appears better fbr CRD estimates with the aggregated CRD 
estimates repealing no clear preference. 

Indicators of Yield Reliability Based <yi 
Direction of Change and Pearson Correlation 
Show the Wllllams-Type Models Preferable for CRD's but CEAS 
Models Better for State and Regional Estimates 
Plots of the aotual and predicted yield for both models as well as the 
strawman model are ahowi for the state level In Figures 6 and 7 . The model 
values and the comparative ranks for the Indicators of yld.d rriiablllty 
based on Y and Y are given In Tables 8, 9, and 10. These indicators 
denonstrate the correspondence between actual and predicted ylrids. The 
better model will ha/e the larger percentage of years In vhlch the direction 
of change from the previous year and from the average of the previous three 
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years In the Y's agrees vdth the corr^ponding change In the Y's. The 
correlatlm coefficient between tiie actual and i»?edicted yields will also 
be hl^er. 

In terms of correct direction of change from the previous year, as 
shovn In Table 8 the Willlams^type nodel ranks at least as g^od as the CEAS 
nndels for all areas except the Minnesota state iiodel and the northeast 
crop reporting district (30) of North Dakota. However, for models aggre- 
gated to the state or regional level, and for the state models, CEAS ranks 
better. Ihe better nodel for each CRD Is shovn In Figure 8. 

Whai the direction of change from the average of the last three years 
Is considered, the Wllllams-type model Is again better than the CEAS models 
for most Individual CRDs (Table 9 )* However, the state and aggregated 
CEAS models are as g^od as, and In the case at Minnesota better than, the . 
Wllllams-type model. The better model for each CRD Is Indicated In Figure 9 - 

In all but three of the CRDs the Pearson correlation coefficient Is 
hl^er for the WUllams-type model (Figure 10 and Table 10). The CEAS 
model has a hl^er correlation coefficient Ibr the Minnesota state model 
and for the Minnesota state estimate derived from aggregation of CRD 
estimates. The CEAS CRD models aggregated to the regional level also hare 
a hl^er correlation coefficient than the WUUanis-type models. 

Statistical Tests Based on d=Y-Y 
Preference for Aggregated CRD Estimate for North Dakota using 

CEAS Model 

Results of the parametric and norqjarametrlc paired-san^sle statistical 
tests are given in Tables 11, 12 and 13. 
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Figure 3. Number Indicates the barley model (a) with smallest percent of test years (1970-1979) having 
absolute value of the relative dlV^erence greater than ten percent. Darker shades indicate 
CRDs with higher production. 
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table 5 

^ ,MODEL^COMPAPlSON BASED ON THE 

percent or YEARS irIlative oifferenqei > i0% 

DERIVED FROM independent TEST YtARS 
TREND AND MONTHLY WEATHER DATA MODELS 

barley 

NORTH DAKOTA AND MINNESOTA 


STATE CRD 1 

STRAWMAN 
% RANK 

1 

1 

williams 
% rank 


CEAS 

RANK 

n.dakcta 10 1 

60 

(2) 

■|*' 

1 

60 

<2) 

40 

(1) 


30 

(1) 

1 

50 

(2) 

70 

(3) 


40 


1 

50 

(3) 


( 1 ) 

40 i 

70 

(2) 

1 

60 

(1) 

80 

(3) 

59 » 

70 

(2) 

1 

60 

<1) 

70 

(2) 

§9 • 

70 

(2) 

1 

80 

(3) 

60 

(1) 

Z9 • 

70 

Q) 

! 


(1) 

60 

(2) 

80 I 

60 

(2) 

1 

50 

(1) 

90 

(3) 

90 1 

I 

60 

(1) 

1 

1 

80 

(3) 

60 

(1) 

state MODEL 1 

40 


1 

1 

80 

(3) 

70 

<2) 

CRDS A6GR.I 

50 

(2) 

1 

70 

(3) 

30 

(1) 

MINNESOTA 10 | 

40 


1 

1 

60 

(3) 

50 

(2) 

40 1 
1 

60 

(3) 

1 

1 

30 

(1) 

50 

(2) 

STATE MODEL 1 

so 

(2) 

1 

1 

50 

(2) 

30 

( } ) 

CRDS AGGR.I 
1 

50 

(2) 

1 

1 

50 

(2) 

30 

(1) 

REGION 1 

CRDS AGGR.I 

50 

(2) 

1 

1 

1 

70 

(3) 

20 

(1) 

STATES AGGR.I 

50 

(1) 

1 

70 

(2) 

70 

(2) 
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dicates the barley model with smalles t value of the largest absolute r 
e test years 1970-1979. Darker shades Indicate CRDs with higJier prod 
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table 6 

model comparison based on the 

TREND and monthly WEATHER DATA MODELS 

Barley 


STATE CRO I 


NORTH DAKOTA AND MINNESOTA 
MODEL 

STRAWMAN i WILLIAMS 

RO rank I RO RANK 


CEAS 

RD RANK 


N. DAKOTA 


10 

20 

30 

AO 

SO 

60 

70 


STAT 


ATE MODEL j 
CROS AGGR.i 


MINNESOTAIO 

AO 

STATE MODEL 
CROS AGGR. 

REGION 

CROS AGGR. 
STATES AGGR. 


65.9 

77.9 
5A.7 
86.3 

73.2 
32.6 

65.2 

51.0 

51.0 


I! 


• 2 (3) 


26.0 (3) 

25. A (2) 


A2.2 

Al.3 


(3) 

(3) 


•26. A 
A6.A 

M 

AO. 7 
•27.6 
•2A.9 
38.6 
•36.5 


•27.7 

33.1 

•2A.8 

•26.9 




7.9 

2.7 


( 1 ) 

( 1 ) 

(!) 

( 1 ) 

( 1 ) 

( 1 ) 

( 2 ) 


■U:t ii! 


( 1 ) 

( 2 ) 

(3) 


( 1 ) 

( 2 ) 


Aa,9 

68.9 

38.5 

71.8 

A1.5 

29.7 

A6.A 

lI:! 

§ 1 :! 


•IA.2 
• 38.6 

•21.5 

•19.3 


2A.7 

•19.2 


( 2 ) 

( 2 ) 

( 2 ) 

( 2 ) 

( 2 ) 

(3) 

( 1 ) 

( 2 > 

( 2 ) 


(2' 

(!) 


( 2 ) 

( 1 ) 
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Figure 5. Nunber Indicates the barley model «lth smallest value of the next largest absolute relative 

difference during the test years 1970-1979, Darker shades indicate CRDs with higher production. 
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table 7 

MODEL COMPARISON 9ASED ON THE 

NEXT largest irelative cifferencei 

DERIVED FROM INDEPENDENT TEST YEARS 
TREND AND MONTHLY WEATHER DATA MODELS 

barley 

north DAKOTA AND MINNESOTA 


MODEL 


STATE CRD 1 

strawman 

PO RANK 

williams 

RD RANK 

CEAS 

RO rank 

N.OAKOTA 10 1 

3A.9 

(3) 

26.8 

(1) 

30.7 

(2) 

20 t 

29.3 

<3) 

-18.3 

(1) 

21.5 

(2) 

30 1 

-15,1 

(2) 

-22.5 

(3) 

lA.A 

(1) 

AO 1 

26.3 


-26.: 

(2) 

-38.1 

(3) 

50 1 

50.3 

(3) 

-20.8 

(1) 

36.6 

12) 

60 1 

28.0 

(3) 

-2C.1 

(1) 

25.0 

(2) 

70 t 

30.2 

(3) 

-17.9 


-28.5 

(2) 

80 1 

A1.7 

(3) 

36.2 

(2) 

-26.6 


90 1 
1 

35.1 

(3) 

-18.8 

(1) 

-27.9 

(2) 

STATE MODEL i 
CRDS AGGR.I 

-1A.5 


-20.1 

(3) 

-19.1 

(2) 

-1A.5 


-19.1 

(3) 

13.7 

ni 

MINNESOTAIO I 

21.1 

(2) 

-26.2 

(3) 

-12.9 

36.1 


AO 1 

I 

A1.9 

(3) 

-2A.2 

(1) 

(2) 

STATE MODEL 1 

21.9 

(3) 

-13,0 

(1) 

-17.2 

(2) 

CRDS AGGR.I 

22. A 

(3) 

-1A.6 

(1) 

-16.0 

(2) 

REGION 1 







CRDS AGGR.I 

-13.7 

(2) 

-17.7 

(3> 

-12.7 


STATES AGGR.I 

13.7 

(1) 

-16.2 

(2) 

18.0 

(3) 
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Figure 8 Number Indicates the barley model (s) with largest percent of test years (1970-1979) having 
agreement In direction of change Trora the previous year between predicted and actual yields. 
Darker shades indicate CRDs with higher production. 
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Figure 9 Number Indicates the barley mode\£s) with largest percent of test years (1970-1979) having 
agreement In direction of change ^otn the previous three year average between predicted and 
actual yields. Darker shades Indicate CRDs with higher production, ^ 
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The CEAS and Wllllaros-type nodels are tested against each other vdth 
the paired c-test; the results are given In Table 11. There Is a signifi- 
cant difference In three of the CRDs, The nonparametrlc test (Table 11 and 
Figure 12) shovfs a significant difference In four of the CFDs and In the 
regional level agg:»egated from CRDs. The Wllllanis-type model Is better In 
thosf' cases where there Is a significant dlffermice; the exception Is the 
northwest Cip (10) In Minnesota (Figure 12).. The CRDs viiere ^ CEAS model 
Is better has hl^er prodictlon than the CRDs there the VdlUams-type model 
Is better. The CEAS model Is also better fbr the aggregated regional 
estimate. 

Results comparing each of the models with a linear trend model are 
Included. The results shotti In Table 12 are the basis for comparison of 
the Wllllams-type model and a "strawnan” nodel (Sebaugji, 1980) for barley 
yield. The paired t-tests show ho slgilflc^t differences (Figure 13) and 
the nonparametrlc rank test (Figure 14) shows significant differences In 
two CRDs with the strawman or linear traid model being better In both 
cases. 

In comparison of the CEAS model with the "strawman” model (Table 13 and 
Figure 15-16), only the CRD estimates aggregated to a state estimate for 
North Dakota show any significant difference between the strawman and CEAS 
model estimates using either the paired t-test or the norparametrlc test. 

The CEAS model estimates were better In this case. CEAS models were also 
better than the Williams t^pe estimate fbr this case. 

In sunmary, the CEAS model estimates were significantly better than the 
WUllaras-type model estimates fbr the aggregated estimate for North Dakota. 
In CRDs having significant differences the CEAS model was better in one 
hl^er producing CRD but the Wllllams-type was better for three lower paro- 
duclng CRDs. CEAS was also better fbr the aggr^tlon to the regional 
estimate. 

0RiG!?v;.v r; :.: 
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Figure 10 Nuaber Indicates the barley aodel with the largest correlation coefflclent'jbe tween actual and 
predicted yields over the test years (1970-1979) i Darker shades Indicate CU)s with higher 
production. ^ 
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Figure (Amparison of model 2 and model 3 ^to predict barley yields based on the percent of test 

>oars (1970-1979) with smaller |df » JY-Yl. Number Indicates model with Jarger percent. 
Blank denotes tie. Stars Indicate the level of significance, none (P;^0.10), *(0.05*P^0. 10) , 
** (0. 01< Pj' 0 . 05) , ***(Pi0.01). Darker shades Indicate CRDs with higher production. 
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Figure 13 Comparison of model 1 and model 2 to predict barley yields based on the average of \d\ - \ V-Y \ 

for 1970-1979. Number Indicates model with smaller average Idj. Blank denotes tie. Stars indicate 
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Figure Comparison of model 1 and model 2 to predict barley yields based qn the percent of tcs 

ysnrs (1970-1979) wlt*l smaller id I - ( Y-Y | . Number indicates model with larger percent. 
Hank dent>tes tie. Stars Indicate the level of significance, none (P/0.10), *(C.05<>PlO. 10) 
**(0.01<’ P^O.05) , ***(P10.0l). Darker shades Indicate CRDs with higher .production. 
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Figures i5 Comparison of model 1 and model 3 to predict fcarley yields based on the average of |d 
lY-Yl for 1970-1979. Number Indicates model with Buiallcr average |dl . C Blank denotes tie. 
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Figure 16 Coroparlson of tiioUel 1 and model 3 ^ ^ no predict barley yields based on the percent of test 
ybflitfS (1970-1979) with smaller |d[ = Y-Y . Number indicates model with larger percent. 

B ank denotes tie. Stars indicate level of significance, none (P 0.10)^ *(0.05 P 0.10), 
**(0.01 P 0.05), ***(P 0.01). Darker shades Indicate CRDs with higher production. 
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Models are Equally Objective 

Hie nodels are entirely objective once the variables ha/e been 
selected. Hie fbrm of the trend variable in both nodels and the deten- 
mlnatlixi of the testure and topological variables fbr WllUams-type nodels 
are subjectively determined prior to the operational use of the nodels. 

. The. selectlm of. .variables -.la aomeMhat sore objecthre for.the WUUans-type 
model since there Is no screening of variables other than fixed statistical 
criteria. 


CEAS Models Attempt to Incorporate 
I&iovii Physical Relationships If 
Statistically Slgilflcant 

Both nodels consider basically the same set of meteorological 
variables. The Wllllams-type tries to eliminate the CPD difference through 
use of soil and topological considerations. Meteorological variables are 
Included If and only If they are statistically slgilflcant fbr a specified 
level of slgilf Icance . The CEAS models apply to Individual areas; neigh- 
boring areas may Incorporate different meteorological variables. The 
Will ’ jms-type nodels use the same variables fbr all areas and Inpllcltly 
assume the response Is the same. Both types of nodels consider the sane 
variables fbr trend. The diolce of trends vas subjective. The deter- 
mination of future or present trends Is a potential problem. Models do not 
detect the Influence of extreme climate conditions beyond the range of con- 
dltlons contained In the developmental data. The Impact of short terra con- 
ditions (less than a month) that are not reflected In the nonthly suimay 
statistics and the Inpact of events of shorter time periods cannot be 
assessed. 
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Adequacy depends upon the requirements of the nodel. The nodels can 
provide estimates at the end of each month If coefficients fbr the 
variables fbr climate conditions prior to the end of that month are 
available. If these aire not aBvallable, an assunptlon or forecast of con- 
dltlons throu^ the rest of the season Is necessary. This capability has 
not been tested. The requirements fbr nodels of these types Include a 
history of yield data and chronologically corresponding climate data. Fbr 
Wllllams-type, Information on soil type and texture Is requii^ed. Both 
types requlr*e background Information regarding the calendar of normal and 
anonBlous crop development. The size of the area ever vtilch the yield Is 

s » 

to be estimated nust correspond to the area used to develop the nodal. The 
nodel nilg^t be adequate fbr a different area of another size, but that 
assun^tlon needs to be confirmed or the nodel shovn to be adequate before 
It Is used. The nodels can not assess the Impact of abrupt changes In 
technology. 


' Timeliness of Models 

Estimates can be obtained with the nodels at the end of the month. How 
quickly these are available depends dlr*ectly upon availability of the 
monthly temperature and precipitation data. If tanper*ature and precipita- 
tion data are derd.ved frem the same source as the historic data, yield 
estimates mould not be available from the nodels uitll one month after the 
end of the month of Interest. Estimates of tanperature and precipitation 
for these same areas, but der*lved from a subset of the stations used in the 
historic data base, may be used Instead. Yield estimates using these data 
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can be a/allable about a yteek after the end of the nonth. Timing of the 
an^allabllity of yield eetlmates probably requires use of these substitute 
data fbr nost applications. The effect of substituting these data In simi- 
lar models has been examined (Sakamoto, et al). 

Difference In Post of 
Models Is Negligible 

Data requirements and sophistication of software required to liq^eroent 
the models operationally are Ins Igilf leant. The WUllans-type models do 
use observations from maiiy CRDs simultaneously; this requires nore storage 
In the con^)uter than a model for a single CHD vtilch ml^t lead to higher 
costs. However, the same model Is used for each CRD vhlch ml^t also 
result In a savings. . 

Models are Simple Ragreaelon Models 

Both models are straightforward application' of ordinary least squares 
regresslm models. Specif Icatlm of the algorithms to calculate some of 
the variables Is the main complication. These ar*e necessary but aice 
Incorporated Into the software should not be any problem. Application 
would be difficult on most hand caLlculators. 

Models Have Poor Current 
Measure of Modeled Yield Reliability 

The Speannan correlation coefficient between the estimate of the stan- 
dard error of a predicted yleid from the base period model, Sy, and the 
absolute value of the difference between the predicted and actual yield, d. 
Indicates vhether the nodel fMTCVldes an accurate current measure of modeled 
yield reliability. It Is desirable to ha/e the r value as close to +1 as 
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possible » Indicating that a narrow confidence Interval (a fmction of Sy) 
about the predicted value is associated with analler deviations of the pre- 
dicted and actual yields. It would be reassuring If the years showing 
large deviations were the years ytien large deviations were expected. 

Examination of Table 14 repeals that not only were the values of the 
correlation not close to +1, but also they were nore often negative. The 


CEAS model was. worse, but neither the CEAS model nor the Williams-type 


V • 


model was e^en close to being acceptable. 


Conclusions 

Models were quite comparable with respect to most of the criteria: 
adequacy, timeliness, cost, slnqplicity and accurate current measures of 
modeled yield reliability. CEAS models are some\^at better with regard to 
requiring scientific and pl^yslcal Interpretation of variables sheeted fbr 
models rather than relying exclusively on statistical slgilf Icance. 
Objectivity is equal once models are developed. The reliability of the 
yield Indication Is somev^at better with the CEAS model in the higher pro- 
ductlm CRDs and In the aggregated estimates. Estimates fbr individual 
CRDs are better with the Wlllians-type model. 


Reconroendatlons 

Since models were quite comparable the recommendation is difficult. 
Althou^ it was believed that CEAS model was slightly better overall to 
achieve an aggregated estimate, it is recommended that the Willisms-type 
not be eliminated. WUUams-type was better fbr several CPD's and was 
better In 1974, an extremely poor year fbr yield (See Appendix 1). These 
models are quite similar with respect to Input requirements and the results 
of model selection criteria. Farther significant Imprwements are quite 
likely going to require additional Inputs and/or changes In model fotro. 
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R3.R 

-1.6 

-3. 1 

-2.2 



1970 

?6.f) 

P^.b 

21.0 

RR. 7 

-l.H 

-4.1 

-3.3 

STATES 

AGGR. . 

1070 

n.7 

?n.9 

10.2 

20. f.' 

2.2 

• 0.5 

1.3 


• . • 

1071 

?4.7 

. 21.4 

21.9 

20.9 

-l.J 

-2.8 

-3.8 



107? 

?1 .0 

?R.6 

22. S 

Rl.r 

0.7 

0.6 

-0.1 



1073 

Rl.l 

23. n 

18.7 

10.5 

1.0 

-2.4 

-1.6 



1 974 

16.7 


U.l 

19.7 

6. Y 

i. • 

3.0 



1075 

?0 .4 

?3.? 

17.7 

17.5 

2.5 

-R. 7 

-2.9 



1976 

R'>.9 

23.3 

17. s 

l“.-i 


-3.3 

-2.6 



1077 

??.9 

2.1.4 

17.7 

1 0. 5 

J.6 

-5.2 

-4.4 



1078 

R'^.3 

R3.>» 

21 

R1 .2 

-1.5 

-3.9 

-4.1 



1070 

RS.O 

?4.5 

R1 .7 

21.6 

-1 .4 

-4.2 

-4.3 


